Thanks for digging into this — the .proj() fix on the antijoin is a sound defensive pattern and the right thing to do regardless.

One question on the motivation: I traced through the code and with the current test fixture (Experiment with Subject as parent), key_source.proj() yields {subject_id} and self (Experiment) shares only subject_id as a common attribute name. The antijoin already matches on PK only in this case, with or without .proj(). Could you share a concrete example (table definitions + reproduce steps) where key_source - self actually returns wrong results today? I'd like to understand the exact conditions — e.g., a custom key_source that retains secondary attributes, or a naming collision between key_source attributes and target secondary attributes.

On the CI failures — two issues to fix:

1. Test logic: Experiment.make() inserts fake_experiments_per_subject = 5 rows per key, so populate(max_calls=2) processes 2 subjects and produces 10 rows, not 2. The assertions need to account for this: assert len(experiment) == 2 * experiment.fake_experiments_per_subject.

2. Lint: ruff-format wants the final assert in test_populate_antijoin_with_secondary_attrs collapsed to a single line. Running pre-commit run --all-files locally will fix it.

Happy to help get these sorted if you'd like — just let me know.

Thanks for the quick response. Excuse the oversight, it was the end of a long day :) I updated everything to reflect the bug.

The exact conditions that trigger the bug:

The antijoin key_source - self matches on ALL common column names. When key_source returns secondary attributes that share names with the target's secondary attributes, SQL matches on those too. If values
differ, no match is found and all keys appear "pending."

Here are the table definitions from our pipeline:

 class LightningPoseOutput(dj.Computed):
     definition = """
     -> LightningPoseTask
     ---
     lp_output_path : varchar(512)
     processing_datetime : datetime
     num_frames = null : int unsigned
     quality = 'good' : enum('good', 'review', 'exclude')
     ...
     """

 class InitialContainer(dj.Computed):
     definition = """
     -> LightningPoseOutput
     ---
     container_path : varchar(512)
     processing_datetime : datetime          # same name, different value
     num_frames : int unsigned               # same name, same value
     quality = 'good' : enum('good', 'review', 'exclude')  # same name, same value
     ...
     """

     @property
     def key_source(self):
         return LightningPoseOutput() & production_models

@Property
def key_source(self):
return LightningPoseOutput() & production_models

When populate() computes pending keys:

• key_source = LightningPoseOutput() → includes processing_datetime, num_frames, quality
• self = InitialContainer() → also has processing_datetime, num_frames, quality
• key_source - self matches on ALL common columns including those 3 secondary attrs
• Since LP's processing_datetime ≠ Container's processing_datetime, no match → all keys appear "pending"

Why we designed it this way:

In this case, InitialContainer simply converts the Lightning Pose CSV output into a https://github.com/neuroinformatics-unit/movement xarray container (NetCDF). The underlying data is the same — same video,
same frames, same quality assessment — so the metadata columns (num_frames, quality, processing_datetime) naturally share the same names. The only value that actually differs is processing_datetime, which
records when each step ran. Having the same column names across parent and child tables is a natural result of a pipeline where each stage wraps or transforms the previous stage's output, and populate() should
determine pending keys based on primary key identity regardless.

That said — if there's a recommended DataJoint pattern for this kind of multi-stage pipeline, let me know :)

Thanks for the detailed explanation — the bug is clear now and it's a real issue.

Caveat: This only triggers when two conditions coincide: (1) a custom key_source that returns secondary attributes (the default key_source already calls .proj()), and (2) the child table has secondary attributes with the same names as the parent's. In your case, LightningPoseOutput and InitialContainer both have processing_datetime, num_frames, and quality, and the custom key_source returns the full LightningPoseOutput() with all those secondary attributes exposed.

User-side fix: You can also resolve this by projecting your custom key_source to primary key only:

@property
def key_source(self):
    return (LightningPoseOutput & production_models).proj()

This strips the secondary attributes before they reach the antijoin, matching the behavior of the default key_source. It's good practice for any custom key_source — return only the primary key attributes that define the work units.

On semantic matching: In DataJoint 2.0, attribute lineage tracking would have caught this. The processing_datetime in LightningPoseOutput (recording when LP ran) and processing_datetime in InitialContainer (recording when the container conversion ran) originate from different computational steps and would have different lineages. The antijoin's assert_join_compatibility check would raise an error on the lineage mismatch rather than silently matching on the wrong columns. This is exactly the class of bugs that semantic matching was designed to prevent.

The .proj() fix in this PR is still the right defensive measure — populate() shouldn't rely on users remembering to project their key_source. CI is green, the regression test is solid. Looks good to merge.

Follow-up on semantic matching: I mentioned that DataJoint 2.0's semantic matching would have caught this, but it turns out it depends on whether lineage tracking is active in your schema.

Semantic matching relies on the ~lineage table, which stores attribute origins. If your schema was created before lineage tracking was added (or schema.rebuild_lineage() was never run), the ~lineage table won't exist. In that case, assert_join_compatibility silently degrades — it logs a warning and skips the check entirely (condition.py:256):

if not expr1.heading.lineage_available or not expr2.heading.lineage_available:
    logger.warning("Semantic check disabled: ~lineage table not found. ...")
    return  # skips the check

So if your pipeline's schemas don't have ~lineage tables, the semantic check was never enforced, and the overlapping processing_datetime attributes were silently matched by the antijoin without any error or warning about lineage conflicts.

To enable semantic matching on existing schemas: schema.rebuild_lineage(). After that, the antijoin key_source - self would raise a DataJointError on the lineage mismatch for processing_datetime rather than silently producing wrong results.